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ABSTRACT 

The concept of accountability in education has two 
primary concerns: the responsibility to provide effective educational 
programs and the responsibility to employ efficiently the resources 
allocated for this purpose. These concerns are fundamental to an 
evaluation procedure based on the principle of accountability. The 
establishment in 1°67 of new Federal programs in Bilingual Education 
and Dropout Prevention provided the vehicle for an effort to 
establish accountability principles. Ten critical factors of program 
desion, operation and management which could expand the dimensions o^ 
accountability were identified: community involvement, technical 
assistance, needs assessment, management systems, performance 

performance contracting, staff development, comprehensive 
cost-effectiveness, and program 
evaluation are discussed. (PR) 
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IMPLICATIONS OF ACCOUNTABILITY ?0R EDUCATIONAL PROGRAM EVALUATION 

W< Stanley Kruger 

The concept of accountability in education! from our point of view, 
haa two primary concerns : the responsibility of the Educational Enterprise 
to provide programs which vill effectively develop the human potential of 
a wide variety of client groups within a diversity of service communities; 
and, the responsibility of the Enterprise to efficiently utilize the vari- 
ous resources entrusted to it by the supporting society. The concept em- 
phasizes optimal attainment of objectives in both these areas of concern, 
and maximization of the desired relationship between them. 

An "Impossible Dream"? We think not; although some dreaming is 
involved, because some planning is Involved. Because some planning is 
Involved, some evaluation is involved - and it must be evaluation with a 
mission. 

As the complexities of modern society generate new demands upon 
Education, they also generate new demands upon other social services, in- 
creasing the competition for resources in the public sector. Thus, events 
transpiring in the real world and attitudes of the public mind combine to 
stress greater attention to the necessity and value of public education, 
and a parallel attention to the necessity for lncrestod implementation of 
principles of accountability in the conduct of educational programs. 

It is not sufficient merely to take notice of a need for accountability, 
or to exhort its virtues. Deliberate, systematic, and consistent procedures 
for development, implementation, avaluatlon, and refinement must be vigor- 
ously pursued. The tenets of the concept permit nothing less from those 
who would embrace it. 

Paper reed before the Invitational Conference on Measurement in Education * 
"Educational Measurement in the Urban Complex", University of Chicago, Chicago, 
Illinois, April 27, 1970. Mr. Kruger ia Technical Aaaiatanca Coordinator, 
Division of Plans A Supplementary Centers, Bureau of Elementary A Secondary 
Education, U.S. Office of Education. 
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Most Federal programs in education support projects which are intended 
to ameliorate and resolve critical problems in the field. If this under- 
taking is to be realised, the design, operation, and management of these 
projects must incorporate specific policies and procedures directed toward 
the attainment of accountability objectives; not only for the dtrect effect 
on critical-problem solution, but also for the indirect effect of generation 
of support for c ntinued efforts in critical-problem areas. In the process, 
we need also to demonstrate techniques with high potential for the renova- 
tion and renewal of traditional program areas, and facilitating systems. No- 
where is this need greater than <n the major urban complexes of our Nation. 

With the passage of the 1967 amendment* to the Elementary and Secondary 
Education Act of 1965, two new Federal programs were established; to support 
projects in the areas of Bilingual Education (Title VII) and Dropout preven- 
tion (Title VIII • Section 807). At the time that basic program regulations, 
manuals, and related materials were being prepared, it was decided that these 
programs provided an appropriate vehicle for a concerted effort to establish 
accountability principles in the administration of Federally-supported proj- 
ects in elementary and secondary education, through a focus upon specific 
aspects of project design and management. The new programs would permit 
Office of Education staff members to apply, ftom the beginnirg, many lessons 
learned in the administration of other Federal programs; particularly from 
the experience of Title 111 (Supplementary Centers and Services), This com- 
ponent of the Elementary and Secondary Edication Act, despite its great po- 
tential, has been unable to effectively demonstrate its accomplishments to 
the degree necessary to sustain continuing fimncial support at growth 
levels. 
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Although accountability in Federal education programs la still very 
much in the developmental stage, the intensity of interest and effort is 
accelerating. The Commissioner of Education's operational objectives for 
Fiscal Year 1971, call for accountability concepts to be installed In ten 
discretionary-fund programs and five State-plan programs by June 30, 1971. 

Plans are now being implemented which will augment the efforts of what has 
heretofore been an extremely small staff effort devoted to theory-building, 
policy-and-procedure formulation, materials development, and staff training. 
Proposals are now being solicited, preparatory to contracts being let, which 
will assist in the refinement of conceptual elements and the production of 
program and instructional materials for use by headquarters, regional office, 
State-agency, and local-educational-agency personnel in disseminating and 
installing accountability concepts in identified programs. 

It should be clear that we are considering a particular type of educa- 
tional accountability - program accountability. That is, the responsibility 
of program personnel, whether in a small grant project, a local-school-system 
curriculum area, an entire local school system, a State school system, a Fed- 
eral program, the U.S. Office of Education, or the entire Federal educational 
effort - to produce an optimum level of results with the resources available 
to them. Subsumed in this consideration of program accountability arc other 
accountability considerations current in the educational literature; such as 
"research accountability", "instructional accountability", and "fiscal account- 
ability", aach of which has particular implications for cartain critical fac- 
tors in program accountability. The focue we would hope to realise, however, 
ie upon the performance of specific, rather complex, organ! set ions with spe- 
cific, rather complex, responsibilities. 
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It is Incumbent upon us, of course, to move beyond concept definition, 
statement of purpose, and justification of need. What are the critical fac- 
tors of program design, operation, and management which must receive partic- 
ular attention if greater dimensions of accountability are to materialize? 

We have identified twelve: community Involvement, technical assistance, needs 
assessment, change strategies, management systems, performance objectives, 
performance or program budgeting, performance contracting, staff development, 
comprehensive evaluation, cost effectiveness, and program auditing. Of these 
twelve factors, at least ten are significant in their implications for educa- 
tional program evaluators. Let us quickly review these ten factors: 

- First, community involvement . A minimum base of community support 
must be ascertained before the commencement of any significant program ac- 
tivities. Beyond this, program personnel should explore every possibility 
to involve members of appropriate concerned community groups in program 
planning, operation, and evaluation. This involvesmnt should include partic- 
ipation in policy determination and in such procedural activities as techni- 
cal-skill levels may warrant. A commitment to accountability, then, requires 
early involvement of program evaluators, to assess initial and continuing 
degrees of community- support, and to assess community-personnel capabilities 
in relation to program skill-requirements. The latter activity, in fact, 
becomes, o.n a broader scale, an essential element of another accountability 
factor, that of - 

• technical assistance . Rarly In the planning of program activities, 
the planning staff should undertake a "capability survey". This survey con- 
sists of setting down the major program objectives, activities, and functions 
and assessing existing systtm capabilities to accomplish them. Where capa- 
bilities do not exist, they must be acquired, either as permanent system 
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additions or on a technical-assistance; consultant or performance-conLraci , 
basis. Program eveluatots' skills are needed in the aasea.?uent of existing 
capabilities in a number of task areas, most particularly to assess the exist- 
ing capability to undertake and complete the assessment and comprehensive 
evaluation necessary to determine program objectives, the extent of program- 
objectives realisation, and the levels of performance efficiency. 

- In needs assessmen t , assuming that basic assessment capability nas 
been determined or acquired, early planning activities will include a basic 
assessment of target-group (client) and situational factors, leading to the 
establishment of task objectives and strategy goals designed to meet needs 
in the respective areas. This activity tckes place, of course, in a climate 
of existing value systems and policies which may also require review in the 
total needs-assessment process. The acquisition of basic decision-making 
data in this phase of program planning is extremely critical, and will demand 
the skill and resourcefulness of able evaluation personnel. 

- Next we determine appropriate management systems . A variety of progrer# 
design and management techniques have been developed in business and industry 
which are now being adapted for use in the educational field. These techniques 
include Program Evaluation and Review Techniques, Critical Path Method, Program 
Planning and Budgeting Systems, Management By objectives, etc. Most deal with 
resource allocation of time, personnel, and funds, in relation to specified 
elements of the primary tasks to be accomplished through program activities. 

The decision to employ any of these systems dictates certain functions, ar- 
rangements, and resource requirements for program evaluation and, thus, in- 
volvement for program evaluators. Within tl.e schema of the selected manage- 
ment systems, program planning proceeds to the specification of • 
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- performance objectives . The keystone of program accountability is 
the statement of program objectives in performance terms, to include the 
nature of performance expected, the direction or level of performance accom- 
plishment expected, the units of performance measurement and means of accom- 
plishing the measurement, and the primary conditions under which performance 
is expected to be conducted. Objectives must be specified for each program 
component, activity or function; by target-group (client); and, in immediate 
to long-range classification. Assistance to program staff in framing per- 
formance objectives in measurable terms is a primary program evaluator 
funccion. 

- Now, performance contracting . Where the decision has been made to 
add program capability by other than permanent system additions (the most 
desirable procedure, where feasible) for continuing program activities or 
functions, the addition should be through performance 'Mtracting. The em- 
phasis in performance contracting is not on traditional input factors (i.e,, 
dollars to be spent, man-hours to bn committed, supplies and equipment to be 
consumed), but, instead, on output factors - the product element obtained 
upon realisation of performance objectives set forth and accepted by the 
contractor as the condition for compensation. The skills of the evaluator, 
again, are needed to assist in the structuring of performance specifications, 
to assure that they are, in fact, stated in significant, valid, measurable 
terms. Beyond this, the evaluator will be called upon to provide the data 
necessary to determine the potential of proposed services, in relation to the 
potentials of alternative services which might accomplish the defined objec- 
tives, including the potential of existing or traditional services being 
utilised by the system. Finally, the program evaluator oust determine if 
and when performance specifications have been met, so that compensation can 
be made. 
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- A standard but important factor in accountability is that of 
staff development . Increased staff competency is essential to progress 
toward the achievement of objectives. Staff development which should in* 
volve program evaluators may be conducted in a wide variety of areas: the 
standards for waiting performance objectives, the techniques for gathering 
data, the bases for interpreting evaluation reports • to cite a few. 

- Next we examine in more detail the accountability factor of 
comprehensive evaluation . This is, of course, the accountability factor of 
primary concern to program evaluators, and a primary concern for many of us 
who work in the administration of Federal programs. The 1967 Miller report 
on the first*year Title ill proposals found evaluation to be the weakest 
element in the proposals. The 1968 report of the Second National Study of 
PACE found, in a review of 94 planning projects, that, "only 30 .... were 
Judged as having adequate evaluation procedures in the project design ... 
contrasted to 31 projects which gave no evidence wh atsoever of evaluation 
procedures." Of 43 operational projects, "a little over eight percent of 
the projects had made plans and promised to be adequate for evaluation of 
their projecta; about 70 percent ... had done a little, and about 13 percent 
had not bothered with evaluation at all." 

From our own work with Federal programs, ve can only conclude that the 
typical Federally*aupported project has an evaluation process that is: un* 
planned, partial, incompetent, uncoordinated, remote, terminal, net row in 
perspective, and underfunded. Project evaluation has largely been an intui* 
tive process conducted by key project staff on an expediency basis. While 
this may have sufficed, in many instances, to maintain project opirations 
at the minimum level necessary to assure continued receipt of Federal funding, 
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it provided little of the substence needed for: (1) accomplishing analyses 
of program effectiveness or impact, cost-benefit relationships, or cost- 
effectiveness comparisons; (2) creating an impetus for systemic change; or 
(3) disseminating and otherwise promoting the adoption of innovative 
practices. 

Who is responsible for the status quo, which someone has defined as 
"the mess we're in"? Many share the blame. Ultimately, for Federally- 
supported projects, the responsibility rests with the Federal program admin- 
istrators who permit the funding of poorly-designed projects, or who permit 
them to continue under conditions of indeterminate effectiveness. We are 
the ones who have produced the program manuals which, at worst, have made 
no mention of evaluation at all, or which, at best, call for applicants to: 
"A. Where applicable, describe the methods, techniques and 
procedures which will be used to determine the degree to 
which the objectives of the proposed program are achieved. 

B. Describe the instruments to be used to conduct the 
evaluation. 

C. Provide a aeparate estimate of costs for evaluation 
purposes. This amount should be included in the proposed 
budget eummary." 

Instructions like the ones Just quoted provide little guidence for the 
development of en adequate eveluetion plen. Then we have compounded the 
problem in ite initial atataa, in some inatancaa, by permitting proposal 
evaluation-plan review, project negotiation, and project monitoring to 
become the eole responsibility of staff who have extreme difficulty in 
distinguishing validity from reliability, and who My associate a "Bell 
curve" with the styling of a Princess telephone. 
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If our own concern for accountability is legitimate; then, what is 
being done to Improve what has been a situation of extreme delinquency? 

First, we are emphasizing the necessity for completion of an evaluation 
plan prior to the commencement of project operations. The plan is to have 
adequa te scope ; to be based upon the specification of performance objectives 
for all basic program activities. For these performance objectives, the 
plan is to indicate key factors and responsibilities associated with measure* 
roent techniques and instruments, data collection, data analysis, reporting, 
and dissemination. Key evaluation-process events are to be posted to en 
evaluation plan "time-line" preparatory to inclusion in an overall project 
work-activity schedule. This "time-line" provides a ready assessment as to 
whether or not the evaluation process is planned as a continuous activity 
rather than a terminal one. 

The evaluation plan is to provide for evaluation of objectives- 
accomplishment at both the operational and at the management levels and, 
within these levels, for attention to evaluation of both product and pr ocess . 

It can readily be seen that we expect the program evaluator, or better 
the program-evalu.Uor team, to possess skills in both of the broad areas of 
"educational evaluation" and "management-operations analysis". The capability 
survey discussed previously should give careful consideration to the competency 
of existing system resources to perform effectively in both areas of a com- 
prehensive evaluation process. The survey should also explore all possibil- 
ities for close coordination between the project evaluation process and the 
more general evaluation activities of the school system, so ar to maximise 
the effectiveness of both, Written comltmsnte establishing this interrela- 
tionship should be secured, 
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It should be apparent by now that we envision evaluation as a funda- 
mental management function, and the evaluator team as an Integral part of 
the project management team. We understand program evaluation as a functional 
skill., rather than as a program phase. Our approach is intended to be a sys- 
tems approach, with the evaluation process continuously gathering and analyz- 
ing data for reporting through "feedback" or "looping" arrangements to the 
planning process. No longer is "objectivity" a primary characteristic of 
the evaluation process (although consistency with reality is much admired). 

We cannot ask program evaluators to develop the schizophrenia that would be 
required if they were to be "integral" and "objective" simultaneously. Nor 
do we want to promote the procrastination, aloofness, ignorance, disin- 
terest, and superficiality that has too often resulted from undue concern for 
objectivity. Objectivity is a desirable quality in the assessment ot program 
operations; ve believe it can more effectively be acquired through an addi- 
tional program-management activity. 

This design may seem to present a large order. It should be pointed out 
however, that we have been citing essential characteristics or elements of an 
adequate evaluation design; not the detail required for each element. Detail 
requirements and the resources to meet them, are a necessary but separate 
consideration, calling for priority and feasibility "trade-offs" within 
resource constraints. Program evaluation, in the minds of too many evalu- 
ation specialists, is synonymous with controlled variables, matched groups, 
normal distributions, equidistant-interval measures, inferential statistics, 
and .01 levels of significance. Unable to attempt the ideal in an area, they 
often attempt nothing; or deliver after the hour of need has passed. As a 
result, the project director frequently has no basis for decision-making 
when, perhaps, it would have been of tremendous value for him to have had 
some simple "exist - not exist" information. 
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- Let us now move on to c ost - effective ness . We heve chosen to separ- 
ate cost-effectiveness analysis from comprehensive evaluation for special 
emphasis, as we did for needs assessment. It logically follows, of course, 
that indicators of unit ob jectives-achieveBumt, when coupled with indicators 
of unit costs, provide us with the basis for cost-effectiveness analysis. 

We are quick to admit our awareness of the difficulties involved in attri- 
buting performance gains to specific project activities or in allocating 
general costs to those activities. We are approaching this problem through 
stages of "successive approximation". This year, for example, we are at- 
tempting to determine an achievement profile for the cluster of objectives 
specified for a project component or activity area and to relate this benefit 
to the full- time-equivalent participant cost associated with total component 
or activity costs - a relationship of "black boxes", if you will. However, 
since some of our projects within a given program have similar component/ 
activity objectives, we have a basis for rudimentary comparisons of cost- 
benefit profiles for different program approaches within comparable environ- 
ments. In this movement toward definitive cost-effectiveness analyses, the 
program-evaluator team has a prominent role; in determining the cost-benefit 
relationships for the project of immediate concern, and in the development 
of feaL s techniques for valid cost-effectiveness analyses in larger-system 
applications. 

- Finally, the program audit . To more effectively monitor the activities 
of the program-evaluator team and, thus, to act as a performance quality con- 
trol on the evaluation process, we have developed cite concept of the program 
auditor. The program auditor is expected to bring three major qualities to 
program operations: objectivity , expertise , and perspective . The ultimate 
purpose of the program audit is the improvement of program operations through 
the improvement of program evaluation. More specifically, the program auditor. 
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through a planned, p re -determined series of activities, highly dependent 
upon sampling techniques, is to determine the appropriateness of evaluation 
procedures, in design and in operation, and to verify the results of the 
evaluation process, thus giving the results an additional measure of credi- 
bility, The latter aim is, again, a matter of considerable value to the 
beleaguered school system in the modern urban complex, 

I have briefly discussed several accountability factors vith substantial 
implications for the program-evaluator team. A given factor cannot be con- 
sidered Independent of the others; nor are the factors discrete. They are 
ordered in what is to us a logical relationship, approximating the sequence 
in which project planners need give the factors attention. Other arrange- 
ments may be more logically organized and, thus, more meaningful to others. 

We do anticipate changes and refinements in the constitutent elements com- 
prising our accountability model. We do feel, however, that the present 
model has already demonstrated its effectiveness in bringing into focus 
those aspects of program design, operation, and management upon which we 
must all diligently work if we are to achieve greater performance in edu- 
cational programs, accompanied by a better use of valuable resources. 

We have had our share of problems in our efforts re accountability. 

In fact, we have encountered most of the problems associated with attempts 
to bring about systemic change; with our own staff as well as with State 
and local-school-system personnel. 

The inertia of rest is a powerful force. A systematic approach to 
educational program operations, even if its ul.imate benefits are visable, 
requires a persistent effort in advance-planning . precision , and thorough - 
ness far beyond that heretofore found in most educational programs. Many 
who undertake the challenge grow weary enroute, and find contentment in 
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pasting new labels on old practices. Many of our Golden Helmets of Mambrino 
Are really shaving basins) 

To continue with Don Quixote for a few more moments, I would have to say 
that many of us have been sent forth by our Lords, in quest of accountability 
- without a horse or a suit of armor. It is far too easy for upper echelon 
administrators, in the view from the castle tower, to fix upon the prize and 
to be oblivious to the resources and support necessary for its attainment. 

We have recognized this to some extent in our relationship with local educa- 
tional agencies and now provide developmental grants for projects selected 
from preliminary-proposal applications, so that necessary technical assistance 
and supporting services can be available for the development of defensible 
plans of operation. We have also established budgetary allocations for eval- 
uation in operational grants at the level of 10 percent of total budget, with 
an additional two percent for program auditing. 

We have not, however, made equivalent resource provisions in our own 
operations. As a consequence, we have program materials available on some 
accountability factors; nothing on others - leading to an uneven emphasis 
and implementation of the model to date. In many areas, our public relations 
efforts have outstripped our program development efforts. Forthcoming de- 
velopmental contracts will assist in correcting this imbalance. 

Accountability is not without its critics - hence the need for a suit 
of armor. Antagonism toward "thinking through 1 ' a new approach to program 
and project administration is prevelent, as the possibility of change brings 
a perceived threat to established securities. Often project and program 
personnel have interests only in general matters of project design or in a 
narrowly-defined curriculum field, and cannot be "bothered" with those aspects 
of project management, however vital, for which they have little personal 
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concern. Others heve vested interests which may be endangered by a more 
open, more coordinated, more interdependent approach to management. Others 
have legitimate concerns which they are u.table to intellectually reconcile 
with the methodology of accountability - the "there's trouble in the affective 
domain" syndrome. All of these groups may find it to their advantage, from 
time to time, to hurl "New Cult of Efficiency" barbs at the proponents of 
accountability. 

The challenge before us is that of demonstrating that accountability 
can contribute to the ijlflllment of all legitimate concerns, that it should 
expose illegitimate concerns, that it is worth the expenditures of effort 
required, and that it need not be devoid of humanism. Meanwhile, we dare 
not promote accoun:ablllty as a panacea for the many complex problems of 
American Education. These problems will not be solved by a single-minded 
approach. Accountability is Important, of course, and always has been; al- 
though, perhaps, its principles have not fully developed under other desig- 
nations, e.g., "quality" or "effectiveness". If a new designation will bring 
strength to needed emphasis, we b. _ ild welcome a new designation. At the 
same time, we should not discard, in moments of hysteria occasioned by our 
anxiety to be rid of public charges of malfeasance as a profession, other 
valuable concepts, including "innovation", "comprehensiveness", "individual- 
ization", "relevance", and "liberation" - else ve do a great disservice to 
our commitment to all that is truly meant by "Education". 
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